Improvements for credential sync - #1548
Merged
Merged
Conversation
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
josephjclark
commented
Sep 14, 2026
josephjclark
commented
Sep 14, 2026
| const overrides: any = { ...pickedSource, ...assigns }; | ||
| const merged: any = assign({}, target, overrides); | ||
|
|
||
| // null in an override (from source or assigns) is an explicit removal |
Collaborator
Author
There was a problem hiding this comment.
This is an importand diff as it allows us to remove keys in the mege (like next, if removing edges, or credentials, which is a bug I've fixed here)
josephjclark
commented
Sep 14, 2026
| const merged: any = assign({}, target, overrides); | ||
|
|
||
| // null in an override (from source or assigns) is an explicit removal | ||
| for (const key of Object.keys(overrides)) { |
Collaborator
Author
There was a problem hiding this comment.
Actually if I remove this code (because no-oneis passing in nulls anyway) everything seems to pass, apart from the bogus test
I think this is the AI leading me around the horn
Collaborator
Author
|
Ok I think this is alright. I'll run some manual tests in the morning |
This comment was marked as resolved.
This comment was marked as resolved.
josephjclark
added a commit
that referenced
this pull request
Sep 15, 2026
* Worker: don't hang when fetch:plan replies with an error joinRunChannel awaited sendEvent(GET_PLAN) inside an unguarded async callback, so a rejection (now possible since Lightning can reply an error to fetch:plan for adaptor resolution failures) became an unhandled rejection instead of settling the outer promise. The worker's local capacity slot for that run leaked until restart. Wrap the await in try/catch and reject, mirroring the existing error/timeout branches in the same file. * Sync removing stuff (rebased) (#1530) * Worker: better reporting on socket errors (#1505) * Worker: attribute sentry reports to the run that produced them Each run already opened a sentry isolation scope, but engine and socket callbacks fire outside its async context, so breadcrumbs and errors were landing on the global scope and interleaving across concurrent runs. The scope is now held on the run context and re-entered where the reporting actually happens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * carry payload size on step-complete events for better reporting * type fix * add payload size to step complete event * report payload size to sentry * changeset --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * Worker: Fix dataclip serialization (#1507) * Worker: attribute sentry reports to the run that produced them Each run already opened a sentry isolation scope, but engine and socket callbacks fire outside its async context, so breadcrumbs and errors were landing on the global scope and interleaving across concurrent runs. The scope is now held on the run context and re-entered where the reporting actually happens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * carry payload size on step-complete events for better reporting * type fix * report better errors out of sentry * Gate double-encoding fix for output dataclips behind WORKER_NO_STRINGIFY_STATE The worker double-JSON-encodes step output dataclips before sending them to Lightning: once via a manual stringify, then again when phoenix serializes the envelope. That double-encoding is what bloats large dataclips past Lightning's websocket frame limit and kills the connection mid-run. The runtime already sanitizes/clones state before it reaches this point, so the manual stringify is redundant - but skipping it changes the wire format, so it's opt-in via --stringify-state/--no-stringify-state (CLI) or WORKER_NO_STRINGIFY_STATE (env), default false to preserve current behaviour until the matching Lightning-side support (2.19+) is widely deployed. * docs and changelog --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * versions * Workflow: add remove()/isRemoved() to flag removed steps, edges and workflows Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Project: generate delete: true entries from Workflow.removed in to-app-state Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix an issue where removing a workflow with history triggers an error * fix remove workflow * clean up comments * hydrate deleted state * update changeset * mock: support for deletes * test for removing workfow * mock: seed version history * update deploy test * changeset * remove dupe test * remove another dupe * remove another dupe * again --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * CLI: checkout direct from project.yaml (#1533) * cli: start ensuring that v2 spec files can be converted to state Several things wrong * project: fix serialisation issue * fixes * remove stray debug console.log calls Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * little refactor * better logging * test: assert credential linkage and edge structure for new-from-spec deploy Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * add test of scenario 1 * Fix an issue where --name is ignored when loading a project from file * ensure cross deployment works from yaml * fix aliases for new projects * fix the alias fix for new projects from checkout * typing * ensure that projects loaded from state files can be loaded statelessly * changeset * fix test * attempt to fix a deploy to an existing project (#1542) * attempt to fix a deploy to an existing project * fix an issue where the wrong alias gets updated * Checkout project v1 (#1546) * support deploying from a v1 spec yaml A v1 spec (as exported from the app) has no uuids and cross-references everything by key, so fromAppState's uuid matching silently mangled it: credentials were dropped and every edge collapsed onto the first job. Convert it to the v2 spec shape instead and let the v2 parser take it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * create to/from-app-spec serializers * tidy * new tests --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Improvements for credential sync (#1548) * CLI: add --credentials option to project deploy Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * add new credentials helpers * refactor credential utils * update docs * mock: throw for unmatched credential * add tests on credential mapping * fix to project merging which fixess credential and channels bugs * update changesets * revert a bunch of junk * clarify comments * support credentials.yaml and update credential id and alias formatting * add a little validation --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ignore credential errors in sentry (#1550) * versions --------- Co-authored-by: Stuart Corbishley <corbish@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Short Description
With deploy v2 this PR lets you:
Fixes #1537
This also makes a refactor to the core merge code, allowing keys to be removed if not present. I might need to think about this a little more.
To make the feature fit nicely there's a little refactoring of some utils
Examples
Deploy a project spec with only the credentials it actually uses (this is actually the default)
Deploy a project spec, stripping all credentials
Deploy a project spec, mapping
a@gmail.org:credtoservice@openfn.org:credDeploy a project spec, with a credentials.yaml with aliases
Neat!
AI Usage
Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):
You can read more details in our
Responsible AI Policy